Personalized Monitoring of Injury Rehabilitation Through Mobile Device Imaging

ABSTRACT

A method and system of diagnosing a medical condition of a target area of a patient using a mobile device are provided. One or more magnetic field images of a target area of a patient are received. One or more hyperspectral images of the target area of the patient are received. For each of the one or more magnetic field images and the one or more hyperspectral images, a three-dimensional (3D) position of the mobile device is tracked with respect to the target are of the patient. A 3D image of the target area is generated based on the received one or more magnetic field images, one or more hyperspectral images, and the corresponding tracked 3D position of the phone with respect to each image. A medical condition of the target area is diagnosed or monitored based on the generated 3D image.

BACKGROUND Technical Field

The present disclosure generally relates to mobile computing devices,and more particularly, to using mobile computing devices to identify ormonitor a medical condition of a patient.

Description of the Related Art

In recent years, mobile wireless communications have become increasinglypopular. Initial implementations of mobile wireless communications, forexample in the form of cellular telephone networks, supported circuitswitched voice communication services. The carriers developed shortmessage service (SMS) technology to provide text and/or e-mailcommunications via the wireless communication networks. As the wirelesscommunication networks have evolved to provide greater bandwidth andpacket-based services, the industry has developed a variety of dataservices, such as email, web browsing, as well as a variety of servicesusing multimedia message service (MMS) technology. Further, mobiledevices have evolved to include an ever-increasing number of features,including WiFi and/or cellular data network-based internet access,global positioning system (GPS) capability, an accelerometer, agyroscope, one or more cameras, light sensor, rotation vector sensor,gravity sensor, orientation sensor, etc. The advanced features supportan ever-increasing range of uses of the mobile devices, such as webbrowsing, email communication, gaming, etc. As the features andcapabilities of mobile devices are steadily increasing, mobile devicesare rapidly becoming the central computer and communication device formany. The compact form factor of mobile devices allows them to be usedalmost anytime and anywhere.

SUMMARY

According to various exemplary embodiments, a mobile device, anon-transitory computer readable storage medium, and a method areprovided to diagnose a medical condition of a patient. One or moremagnetic field images of a target area of a patient are received. One ormore hyperspectral images of the target area of the patient arereceived. For each of the one or more magnetic field images and one ormore hyperspectral images, a three-dimensional (3D) position of themobile device with respect to the target are of the patient is tracked.A 3D image of the target area is generated based on the received one ormore magnetic field images, one or more hyperspectral images, and thecorresponding tracked 3D position of the phone. A medical condition ofthe target area is diagnosed and/or monitored based on the generated 3Dimage.

In one embodiment, receiving one or more magnetic field images of thetarget area includes emitting a magnetic field by a transceiver of themobile device onto the target area and receiving at least one of radioor magnetic signals from the target area in response to the emittedmagnetic field of the transceiver. The magnetic signal from the targetarea may be received by a magnetic field sensor of the mobile device.Receiving one or more magnetic field images of the target area mayfurther include, for each of the one or more magnetic field images,providing guidance on a user interface of the mobile device as to how toposition the mobile device in 3D space with respect to the target area.

In one embodiment, receiving one or more hyperspectral images of thetarget area includes, for each hyperspectral image, controlling a lightsource of the mobile device, to emit light at one or more predeterminedwavelengths; and recording a hyperspectral image of an anatomy of thetarget area by a camera of the mobile device. Receiving one or morehyperspectral images of the target area may further include, for each ofthe one or more hyperspectral images: providing guidance on the userinterface as to how to position the mobile device in 3D space withrespect to the target area.

In one embodiment, at least one of the one or more magnetic field imagesand at least one of the one or more hyperspectral images are takenconcurrently from a same position in 3D space with respect to the targetarea.

In one embodiment, for each hyperspectral image, a quality of aresolution of the hyperspectral image is determined. Upon determiningthat the quality of the resolution of the hyperspectral image is below apredetermined threshold, the image is enhanced by way of a deep learningmodel.

In one embodiment, the 3D image is further based on one or morephotographs taken by a camera of the mobile device.

According to one embodiment, a computer implemented method includesdirecting a transceiver of a mobile device to emit a magnetic field ontoa target area of a patient. A signal is received in response to theemitted magnetic field. One or more magnetic field images of the targetarea are created. A light emitting source of the mobile device iscontrolled such that light is generated at one or more differentwavelengths. One or more hyperspectral images of the target area arecreated in response to the light generated at the one or more differentwavelengths. A 3D position of the mobile device is tracked with respectto the target area for each of the one or more magnetic field images andthe one or more hyperspectral images. A 3D image of the target area isgenerated based on the created one or more magnetic field images, theone or more hyperspectral images, and the corresponding tracked 3Dposition of the mobile device. A medical condition of the target area isdiagnosed and/or monitored based on the generated 3D image.

These and other features will become apparent from the followingdetailed description of illustrative embodiments thereof, which is to beread in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are of illustrative embodiments. They do not illustrate allembodiments. Other embodiments may be used in addition or instead.Details that may be apparent or unnecessary may be omitted to save spaceor for more effective illustration. Some embodiments may be practicedwith additional components or steps and/or without all of the componentsor steps that are illustrated. When the same numeral appears indifferent drawings, it refers to the same or like components or steps.

FIG. 1 illustrates an example architecture for implementing a systemthat provides medical imaging using a mobile device.

FIG. 2 illustrates a block diagram showing various components of anexample mobile device at a high level, consistent with an illustrativeembodiment.

FIG. 3 is a block diagram of a disease detection system, consistent withan illustrative embodiment.

FIG. 4A is a block diagram of an image acquisition block harvestinginformation from an anatomy of a patient, consistent with anillustrative embodiment.

FIG. 4B illustrates a patient receiving instructions from a mobiledevice to perform a scan of a target area, consistent with anillustrative embodiment.

FIG. 5 presents an illustrative process for identifying a medicalcondition of a target area of a patient by way of a mobile device.

FIG. 6 is a functional block diagram illustration of a computer hardwarethat can communicate with various networked components.

DETAILED DESCRIPTION Overview

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent that the presentteachings may be practiced without such details. In other instances,well-known methods, procedures, components, and/or circuitry have beendescribed at a relatively high-level, without detail, in order to avoidunnecessarily obscuring aspects of the present teachings.

The present disclosure generally relates to medical imaging using amobile device. Injuries or other external or internal health status,collectively referred to herein as a medical condition, can bedetermined by way of using various sensors that are available in amobile device. These sensors can be used to generate images of not justwhat is visible externally on a user (e.g., patient), but also theunderlying bones, tissues, and organs. The images can be used togenerate a three-dimensional rendering of the anatomy of a target areaof a patient.

Any kind of injury affects the quality of life as it limits theindividual's ability to perform routine activities. During the recoveryperiod, depending upon the seriousness of the injury, the patient maywant to regularly consult a medical professional to determine how wellhe or she is recovering. Often visits to a professional medical facilitymay not be practical in various scenarios. For example, regular visitsto a clinic may be inconvenient (e.g., particularly for the elderly);there may be delays in the availability of medical equipment, such asmagnetic resonance imaging (MM) or CAT (CT) scan; there may bescheduling conflicts with daily routines; the procedures may be costprohibitive; etc. During the period between visits to the medicalprofessional, there may be no effective way to determine how well aparticular medical condition (e.g., muscle, bone, tendon, organ,internal infection, etc.) is healing, since specialized medicalequipment (which may only be available at a clinic) may be involved toimage the anatomy of the subject medical condition. Such a situationposes the following challenges. Consequently, a patient's medicalcondition may not progress optimally or worsen by simply being unawareof the progress.

Accordingly, what is discussed herein are methods and systems of usingmobile devices that are particularly configured to provide apersonalized monitoring of a medical condition of a patient. Acombination of sensors that are inherent in modern mobile devices arecontrolled in a specific way to transform the mobile device to be ableto generate images of not just what is externally visible, but also theunderlying bones, tissues, and organs, as well as their state ofrehabilitation. In one embodiment, by combining acquired images usinghyperspectral imaging and magnetic field images generated from thephone, along with the device's orientation in 3-dimensional (3D) space,and applying methods that stitch together features from separate images,it is possible to generate 3D models of the underlying bone, tissue ororgan that provide significantly improved understanding of theunderlying medical condition, and the progress of rehabilitation.

The techniques described herein may be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures.

Example Architecture

FIG. 1 illustrates an example architecture 100 for implementing a systemthat provides medical imaging using a mobile device. Architecture 100includes a network 106 that allows various mobile devices 102(1) to102(n) of corresponding users 101(1) to 101(N), sometimes referred toherein as patients, to communicate with various components that areconnected to the network 106, such as one or more patient database 110,reference database 112, and an authorized medical professional 120. Thenetwork 106 may be, without limitation, a local area network (“LAN”), avirtual private network (“VPN”), a cellular network, the Internet, or acombination thereof. For example, the network 106 may include a mobilenetwork that is communicatively coupled to a private network, sometimesreferred to as an intranet, which provides various ancillary services,such as communication with various application stores, databases, andthe Internet. To facilitate the present discussion, network 106 will bedescribed, by way of example only and not by way of limitation, as amobile network as may be operated by a carrier or service provider toprovide a wide range of mobile communication services and supplementalservices or features to its subscriber customers and associated mobiledevice users 101(1) to 101(N). The network 106 allows users of themobile devices 102(1) to 102(n) to communicate with each other and toreceive data from or provide data to the patient database 110, referencedatabase 112, and/or an authorized medical professional 120.

For purposes of later discussion, several mobile devices appear in thedrawing, to represent some examples of the devices that may receivevarious services via the network 106. Today, mobile device's typicallytake the form of portable handsets, smart-phones, tablet computers,personal digital assistants (PDAs), smart watches, and laptops, althoughthey may be implemented in other form factors, including consumer, andbusiness electronic devices.

A mobile device 102(1) to 102(N) may have various applications stored inits memory that may have been downloaded from various applicationstores. Each mobile device that is subscribed to the medical imagingdescribed herein, includes an application, sometimes referred to hereinas the diagnosis engine, is operative to capture various images of atarget area of a patient. For example, the target area may represent aninjury, such as, without limitation, a skin tear, infection, bonefracture, ruptured tendon, etc. The diagnosis engine can controlhardware that may be inherent in its corresponding mobile device togenerate various signals, including magnetic fields of differentpredetermined strength and light at different wavelengths. The diagnosisengine further configures the mobile device to receive various signalsin response to the emitted magnetic fields and light. The signalsreceived in response to the magnetic fields generated by the mobiledevice are used to create a magnetic field image. In contrast toelectric signals, which are influenced by the differently conductivetissue of the body and varying resistance of the skin before they can berecorded, the magnetic signals travel through the body almost withoutdisturbance, thereby being able to observe structures below the skin ofthe patient.

The control of a light source of the mobile device, such as a lightemitting diode (LED), may be used to create hyperspectral imaging of thetarget are of the patient. The hyperspectral imaging discussed hereinuses a camera of a mobile device to collect and processes informationfrom across an electromagnetic spectrum of the controlled light sourceof the mobile device. In this way, the camera of a mobile device canobtain the spectrum for each pixel in the image of a target area of thepatient, with the purpose of identifying the underlying structure. Forexample, the camera of a mobile device collects information in the formof a set of images. Each image represents a narrow wavelength range ofthe electromagnetic spectrum, also known as a spectral band. Theseimages are combined to form a three-dimensional (x,y,λ) hyperspectraldata array for processing and analysis, where x and y represent twospatial dimensions of the scene, and λ represents the spectral dimensioncomprising a range of wavelengths. The relative position of the mobiledevice in 3D space with respect to the target area is recorded by themobile device to later be able to construct a 3D image from the gatheredimages and corresponding positions.

Accordingly, both the magnetic field images and the hyperspectral imagesmay be taken from different positions, guided by the mobile device. Forexample, the diagnosis engine may interact with a patient database 110to determine the relevant target area of a patient, the range ofmagnetic fields, the light wavelengths, the number of images to take foreach of the magnetic field imaging and hyperspectral imaging, and whichpositions in 3D space in relation to the target area of the patient. Inone embodiment, the diagnosis engine uses various sensors, such asoptical (e.g., camera), accelerometer, and/or gyroscope to provideguidance to the user holding the mobile device as to how to positionand/or move the mobile device in 3D space to be capture thehyperspectral and magnetic field images, respectively. For example, forskin tears or infections, a predetermined set of wavelengths, magneticfields, and/or a number of pictures may be taken, whereas for a bonefracture, a different set of wavelengths, magnetic fields, and/or numberof pictures may be dictated.

In various embodiments, the magnetic field images and the hyperspectralimages may be taken concurrently or separately. For example, thediagnosis engine may first provide guidance to move the mobile device in3D space to complete the requisite magnetic field images from differentpositions while controlling the magnetic field generated. Upondetermining that the magnetic field images are complete, the diagnosisengine stops generating a magnetic field and initiates control of thedesired wavelength of light, while providing guidance to move the mobiledevice in 3D space to complete the requisite hyperspectral images. Inanother embodiment, the order of types of images taken can be reversedor performed concurrently. In one embodiment the magnetic field used andthe wavelengths of light used, based on the condition beinginvestigated, may be provided by the patient database 110, discussed inmore detail below.

In various embodiments, the guidance to position the mobile device in 3Dspace with respect to the target area of the patient may be provided viathe speakers of the mobile device as voice instructions (e.g., “pleasegradually move closer to the elbow while rotating the phone to theleft”), audible tone (e.g., beeps), messages on a display of the mobiledevice, augmented reality on the display of the mobile device, hapticsignals, or any combination thereof.

By combining the acquired images from different positions with respectto the target area of the patient using hyperspectral imaging andmagnetic field images generated from the mobile device, and applyingmethods that stitch together features from separate images, theteachings herein generate 3D models of the underlying bone, tissue,and/or organ, which can provide a significantly improved understandingof the underlying medical condition and the progress of rehabilitation.In some embodiments, the results may be saved in the patient database110 and/or provided to an authorized medical professional 120. Forexample, if the progress is below a predetermined threshold (e.g., notprogressing as reference expected data when compared to expected datareceived from a reference database 112), an electronic message is sentto the authorized medical professional 120, which may include thegenerated 3D image of the target area of the patient.

As mentioned above, the architecture 100 may include a patient database100 that is operative to provide its account holders (e.g., subscribersto the diagnosis engine service discussed herein) on-line access to avariety of information related to a user's (e.g., patient's) account,such as existing medical conditions, medical issues to monitor, specifictarget areas of the patient, and the like. The patient database maymaintain an ongoing history of all prior information related to aninjury being monitored, as well as a database of past scans. Over time,and across a growing user base, the patient database 110 can learn toidentify relative progress of different medical conditions (e.g.,injuries). This learning could for example be in the form of machinelearning where rehabilitation progression is modelled as a function ofmultiple inputs, such as gender, age, degree of injury severity, as wellas any other relevant features that may be correlated to the rate ofrehabilitation. Depending upon the target application and the number ofpatient records, in one embodiment, a random forest classifier/regressor(for less patient data) or a neural network architecture (for largenumber of records) may be used. Further it can provide a mobile device(e.g., 102(1)) of a patient (e.g., 101(1)) and/or an authorized medicalprofessional 120 various information, such as, without limitation, (i) acondition/disease affecting the area; (ii) a severity score of thedisease; (iii) a change in status compared to a previous scan; (iv)disease progression statistics compared to other patients having asimilar condition; (v) recommended future steps (e.g., whether and whento visit a medical professional). In one embodiment, the features of thepatient database 100 are part of the diagnosis engine of the mobiledevice.

While the patient database 110 and reference database 112 have beenillustrated by way of example to be on different platforms, it will beunderstood that in various embodiments, their functionality describedherein can be combined or even be part of a mobile device. In otherembodiments, these computing platforms may be implemented by virtualcomputing devices in the form of virtual machines or software containersthat are hosted in a cloud, thereby providing an elastic architecturefor processing and storage. Each of the databases and computing devicesdiscussed herein are compliant with the Health Insurance Portability andAccountability Act (HIPAA), which sets the standard for protectingsensitive patient data.

Example Mobile Device

As discussed in the context of FIG. 1, the determination of a medicalcondition of a patient with respect to a target area involves the use ofa mobile device. In this regard, FIG. 2 illustrates a block diagramshowing various components of an illustrative mobile device 200 at ahigh level. For discussion purposes, the illustration shows the mobiledevice 200 in the form of a smartphone device, while it will beunderstood that other mobile computing devices are contemplated as well.

The mobile device 200 may include one or more antennae 202; atransceiver 204 for cellular, Wi-Fi communication, short-rangecommunication technology, and/or wired communication; a user interface206; one or more processors 208; hardware 210; and memory 230. In someembodiments, the antennae 202 may include an uplink antenna that sendsradio signals to a base station, and a downlink antenna that receivesradio signals from the base station. In some other embodiments, a singleantenna may both send and receive radio signals. The same or otherantennas may be used for Wi-Fi communication and the receipt of magneticor radio signals that are the response of the body to magnetic fieldsfor magnetic field imaging. These signals may be processed by thetransceiver 204, sometimes collectively referred to as a networkinterface, which is configured to receive and transmit digital data.

In one embodiment, the mobile device 200 includes one or more userinterface(s) 206 that enables a user to provide input and receive outputfrom the mobile device 200. For example, the user interface 206 mayinclude a data output device (e.g., visual display(s), audio speakers,haptic device, etc.) that may be used to provide guidance to a user ofthe mobile device 200 such that the mobile device 200 is properlypositioned in 3D space with respect to a target area of a patient. Theuser interface 206 can also be used to display a representation of abone, tissue, organ, etc., of the patient as well as a diagnosis of anidentified injury based on the 3D representation.

The user interface(s) 206 may also include one or more data inputdevices. The data input devices may include, but are not limited to,combinations of one or more of keypads, knobs/controls, keyboards, touchscreens, microphones, speech recognition packages, and any othersuitable devices or other electronic/software selection interfaces.

The mobile device 200 may include one or more processors 208, which maybe a single-core processor, a multi-core processor, a complexinstruction set computing (CISC) processor, gaming processor, or anyother type of suitable processor.

The hardware 210 may include a power source and digital signalprocessors (DSPs), which may include single-core or multiple-coreprocessors. The hardware 210 may also include network processors thatmanage high-speed communication interfaces, including communicationinterfaces that interact with peripheral components. The networkprocessors and the peripheral components may be linked by switchingfabric. The hardware 210 may include hardware decoders and encoders, anetwork interface controller, and/or a USB controller.

The hardware 210 may include various sensors to determine theorientation/position of the mobile device 200. For example, there may beone or more accelerometers 212 that are configured to measureacceleration forces, which may be used to determine an orientation ofthe mobile device 200. There may be a gyroscope 214, which allows themeasure of the rotation of the mobile device, as well as lateralmovements. The accelerometer(s) 212 and the gyroscope 214 may be usedtogether to provide guidance as to how to position and the speed of themovement of the mobile device 200.

The hardware 210 may further include a GPS sensor 216 that is operativeto provide a location of the mobile device and its speed. In oneembodiment, the geographic location, which may include altitudeinformation, may be used to better estimate the expected progress of amedical condition. For example, climate and altitude may be salient inthe healing process.

The hardware 210 may include one or more cameras 218 that are operativeto take photographs under different lighting conditions, which may beprovided at least in part by the LED 222 of the mobile device 200 tocapture regular and/or hyperspectral images of a target area of apatient. The one or more cameras 218 may also be used together with theaccelerometer(s) 212 and the gyroscope 214 to guide the mobile device200 to the appropriate position in a 3D space with respect to a targetarea of the patient. For example, an augmented reality image may bedisplayed on the user interface with instructions on how to maneuver themobile device. The resulting images can then be stored in a memory 230of the mobile device 200 and/or shared with different recipients, suchas a patient database, reference database, and/or authorized medicalprofessional, based on permission settings of the diagnosis engine 242.

Today, mobile devices typically include a magnetic field sensor 220. Forexample, the magnetic field sensor 220 may be a small-scalemicroelectromechanical systems (MEMS) device for detecting and measuringmagnetic fields. Such magnetic sensor may measure the effects of theLorentz force.

The mobile device 200 includes a memory 230 that may be implementedusing computer-readable media, such as computer storage media. Storagemedia includes volatile and non-volatile, removable and non-removablemedia implemented in any method or technology for storage of informationsuch as computer-readable instructions, data structures, programmodules, or other data. Computer storage media includes, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD), high definition video storagedisks, or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othernon-transmission medium that can be used to store information for accessby a computing device.

The memory 230 may store various software components or modules that areexecutable or accessible by the processor(s) 208 and controller(s) ofthe mobile device 200. The various components of the memory 230 mayinclude software 232 and an operating system 250. The software 232 mayinclude various applications 240, such as a diagnosis engine 242 havingseveral modules, each configured to control a different aspect of thedetermination of a medical condition of a subject area of a patient.Each module may include routines, program instructions, objects, and/ordata structures that perform tasks or implement abstract data types. Forexample, there may be a magnetic field image module 243 operative togenerating various magnetic field images of a target area of a patient.For example, magnetic field image module 243 of the diagnosis engine 242may direct the transceiver 204 of the mobile device 200 to emit a weakmagnetic field and receive signals in response thereto to create aninitial magnetic field image of the target area of the patient. Theseresponse signals may be radio and/or magnetic signals resulting from theresponse of the body to the magnetic field, and can be measured afterthe field is no longer being applied. The resulting magnetic field imagemay be of a relatively low quality due to the weak field strength. Itsrepresentation and quality are improved using the quality enhancementmodule described later. In some embodiments, the resulting magneticfield is measured by the magnetic field sensor 220 of the mobile device200.

The diagnosis engine 242 may include a hyperspectral image module 244that is operative to generate various hyperspectral images of a targetarea. In one embodiment, the hyperspectral image module 244 is operativeto control the LED 222 light source of the mobile device 200 to generatelight at different wavelength to obtain a hyperspectral image of theanatomy being monitored, sometimes referred to herein as the targetarea. The hyperspectral image module 244 interacts with the camera 218to capture the hyperspectral images of the target area.

The diagnosis engine 242 may include an image reconstruction module 245that is operative to combine the one or more magnetic field images fromthe magnetic field image module 243 and one or more hyperspectral imagesprovided by the hyperspectral image module, and the 3D positioning datato generate a detailed representation of the anatomy with a sufficientamount of detail to be able to discern the current condition of theskin, tissues and/or bone to be able to later diagnose a medicalcondition of a patient based on the detailed representation. Forexample, for each of the magnetic field and hyperspectral images, thevarious sensors discussed herein, such as the accelerometer 212 and thegyroscope 214, provide coordinates with respect to the target area. The3D position information together with the images are used to generate a3D representation. In one embodiment, each of the one or more magneticfield images and one or more hyperspectral images will comprise a pointcloud in which each pixel of the image has an associated set of x, y,and z coordinates in 3D space. To stitch two images together, differentmethods for this purpose can be applied. For example, scan matching orparticle filtering (as used in robotics), or another similar techniquecan be applied. In these techniques, the change in 3D position measuredby the accelerometer 212 or the gyroscope 214 can be used as a firstestimate for the relative positions of the two-point clouds of twoimages. Thereafter, via several iterations of either of thesetechniques, the true relative difference in 3D position between twopoint clouds can be established more exactly, and the two point cloudscan be merged into a single, larger point cloud. The point cloud ofevery subsequent image can be stitched into this single growing pointcloud in the same manner. In some embodiments, the image reconstructionmodule 245 is operative to further enhance the 3D representation of thetarget area by way of machine learning algorithms, discussed in moredetail later.

In one embodiment, there is a 3D positioning module 246 that isoperative to determine a position of the mobile device 200 with respectto a target area of a patient. To that end, the 3D positioning module246 uses various sensors of the mobile device 200, such as theaccelerometer 212, gyroscope 214, and/or camera 218 to determine aposition of the mobile device 200 in 3D space with reference to thetarget area of a patient. For example, at time t0, the mobile device 200acquires one magnetic and one hyperspectral image. At time t1, oneadditional magnetic and one additional hyperspectral image aregenerated. By virtue of the positioning module 246, the mobile device200 is able to track its position (e.g., between t0 and time t1, itmoved x degrees around its x-axis, y degrees around its y-axis, and zdegrees around its z-axis).

In one embodiment, the diagnosis engine 242 includes a guidance module247 that is operative to provide guidance as to how to position themobile device 200 such that it is positioned properly for the capturingof the magnetic field and hyperspectral images. To that end, theguidance module 247 may provide instructions via various hardware 210components of the mobile device 200, such as the speakers, messages on adisplay (e.g., user interface 206), augmented reality on the display,haptic signals, or any combination thereof.

In one embodiment, the diagnosis engine 242 includes an image analysismodule 248 that is operative to receive the 3D representation of thetarget area from the image reconstruction module 245 and identify (i.e.,diagnose) a status of the medical condition of the target area based onthe same. In various embodiments, the diagnosis engine 242 may beindependent, or may work together with the patient database 110 and/orreference database 112 discussed before in the context of thearchitecture 100 of FIG. 1. For example, the image analysis module 248determines the status of the medical condition of the target area of thepatient, performs comparisons between the present state (i.e., medicalcondition), the previous state, and a reference state (e.g., expectedpresent state based on extrapolations) and provides recommendations(e.g., via the user interface 206 of the mobile device 200).

The operating system 250 may include components that enable the mobiledevice 200 to receive and transmit data via various interfaces (e.g.,user controls, communication interface, and/or memory input/outputdevices), as well as process data using the processor(s) 208 to generateoutput. The operating system 250 may include a presentation componentthat presents the output (e.g., display the data on an electronicdisplay of the mobile device 200, store the data in memory 230, transmitthe data to another electronic device, etc.). Additionally, theoperating system 250 may include other components that perform variousadditional functions generally associated with an operating system 250.By virtue of the hardware and software of the mobile device 200, thediagnosis engine 242 transforms the mobile device into an efficientportable medical imaging system and diagnosis device.

Example Block Diagrams

Reference now is made to FIG. 3, which is a block diagram of a diseasedetection system 300, consistent with an illustrative embodiment. Thesystem 300 includes an image acquisition block 310 that is coupled to aquality assessment block 320. There is a disease detection block 330coupled to the quality assessment block. There is a disease progressionanalysis block 340 coupled to the output of the disease detection block330 and operative to provide an output to a recommendation block 350.

The image acquisition block 310 is operative to capture various imagesof a target area of a patient by way of a mobile device. The images maybe based on magnetic field images and/or hyperspectral images that weretaken from different 3D positions relative to a target area of apatient. For example, a diagnosis engine directs a user to move themobile device to different positions, while inducing a magnetic fieldand/or light wavelength at various regions of a target area of apatient. These images can be regarded as initial magnetic field and/orinitial hyperspectral images.

In one embodiment, at block 320, a quality assessment is performed ofeach of the images. For example, for each hyperspectral image, thediagnosis engine determines a quality of a resolution of thehyperspectral image. Upon determining that the quality of the resolutionof the hyperspectral image is below a predetermined threshold, the imageis enhanced by way of a deep learning model. If the resolution of theimage cannot be enhanced, the mobile device is guided to an appropriatelocation in 3D space with respect to the target area, to harvestadditional images. A similar approach may be used for each magneticfield image.

The magnetic field image generated by the phone's hardware may be ofrelatively low quality as compared to professional equipment, since thegenerated magnetic strength of the transceiver (e.g., in the order of1000 nT to 6000 nT) is much lower than those used in magnetic resonanceimaging machines (greater than 1 T). Upon determining that the imageresolution is below a predetermined threshold, to improve image quality,an artificial intelligence module may be used for image enhancement. Inone embodiment, the enhancement module conducts image super resolutionand quality enhancement directly in an unsupervised manner to improveimage resolution and quality. In another embodiment, the enhancementmodule is based on a convolutional neural network (CNN) that haspreviously been trained using as training inputs a set of similarlyobtained magnetic images, and using as training outputs a set of imagesobtained using professional equipment. When new low-quality images areobtained, they can be run through this previously trained network tofind the previously obtained high quality image that is the closestmatch. The resulting high quality image may be chosen from a set ofpre-existing high quality images, or may be generated usingsuperposition of multiple pre-existing high quality images.Convolutional neural networks and generative adversarial networks arepart of the deep learning architecture.

Upon determining that the quality of each image is above a predeterminedthreshold (or bringing the quality of each picture to above thepredetermined threshold as discussed above) a 3D image of the targetarea is generated based on the one or more magnetic field images and oneor more hyperspectral images (which may have been enhanced). In someembodiments, the 3D image generated is further improved usingphotographs taken by a camera of the mobile device. For example, thephotographs may only show the visual exterior of the body part ofinterest and do not contribute to the interior structure of the 3Dimage, but they can be used to provide a “skin” for the image whichhelps to identify how the underlying 3D structure maps to the area ofthe body under consideration.

At block 330, the generated 3D image is used to diagnose a medicalcondition (e.g., disease) of the target area. For example, the image maybe compared to historic data of different diseases. If the patternidentified in the 3D image is sufficiently similar to one provided inthe historical data, then the 3D image is deemed to be consistent withthe medical condition of the historical data. In one embodiment, themeasure of similarity is conducted by using a deep learning approach inwhich a neural network has been trained to take as input a 3D image andto output the most likely medical condition, using previously labelleddatasets. In another embodiment, a clustering analysis is conducted togroup many scans from multiple patients showing similar features into adiscrete set of groups that can be labelled as corresponding to aparticular condition.

In one embodiment, upon determining the medical condition, at thedisease progression analysis block 340, the data of the 3D image iscompared to previous scans (e.g., 3D images) to determine the progressin rehabilitation (or lack thereof). There is a recommendation block 350that may provide feedback on a user interface of the mobile device as tofurther actions to take in order to improve the identified medicalcondition.

Reference now is made to FIG. 4A, which is a block diagram 400A of animage acquisition block 420 harvesting information from an anatomy 410of a patient, consistent with an illustrative embodiment. By way ofexample and not limitation, the anatomy 410 may be a broken arm of apatient 400B, who is guided by the mobile device 404 to perform a scanof the target area, as illustrated in FIG. 4B. The image acquisitionblock 420 may include a magnetic field image block 422, a hyperspectralimage block 424, a 3D positioning block 426, and an output image block428. The anatomy 410 of a patient is scanned at a target area by themagnetic field image block 422 and the hyperspectral image block 424. Asdiscussed herein, the magnetic field image block 422 emits a magneticfield and receives signals in response thereto. The response signals areused to create a magnetic field image of the target area of the patient.Similarly, the hyperspectral image block 424 controls a light source ofthe mobile device 404 to generate light at different wavelength toobtain a hyperspectral image of the anatomy 410 of the target area. Themagnetic field images of the magnetic field image block 422, as well asthe hyperspectral images of the hyperspectral image block 424 may beharvested from different positions in 3D space with respect to thetarget area of the patient.

The 3D positioning block 426 uses various sensors discussed herein torecord the coordinates (i.e., 3D position in space with respect to thetarget area of the patient) for each image captured. In this way, theoutput image block 428 is able to create a 3D image of the anatomy 410of the patient by combining the information from the magnetic fieldimage block 422, hyperspectral image block 424, and the 3D positioningblock 426.

Example Process

With the foregoing overview of the architecture 100, example mobiledevice 200, and example disease detection system 300, it may be helpfulnow to consider a high-level discussion of an example process. To thatend, FIG. 5 presents an illustrative process 500 for identifying amedical condition of a target area of a patient by way of a mobiledevice. Process 500 is illustrated as a collection of blocks in aprocess, representing a sequence of operations that can be implementedin hardware, software, or a combination thereof. In the context ofsoftware, the blocks represent computer-executable instructions that,when executed by one or more processors, perform the recited operations.Generally, computer-executable instructions may include routines,programs, objects, components, data structures, and the like thatperform particular functions or implement particular abstract datatypes. The order in which the operations are described is not intendedto be construed as a limitation, and any number of the described blockscan be combined in any order and/or performed in parallel to implementthe process. For discussion purposes, the process 500 is described withreference to the architecture 100 of FIG. 1 and the mobile device 200 ofFIG. 2.

At block 502, a diagnosis engine 242 of a mobile device 200 receives oneor more magnetic field images of a target area of a patient. To thatend, the diagnosis engine 242 controls a transceiver 204 of the mobiledevice 200 such that a magnetic field is emitted onto the target area.Radio and/or magnetic field signals are received from the target area inresponse to the emitted magnetic field of the transceiver 204. In oneembodiment, the magnetic field signal from the target area is receivedby a magnetic field sensor 220 of the mobile device 200.

At block 504, one or more hyperspectral images of the target area of thepatient are received by the mobile device 200. To that end, thediagnosis engine 242 controls a light source of the mobile device, suchas an LED 222, such that light at one or more predetermine wavelengthsis generated. A hyperspectral image of an anatomy of the target area isreceived by a camera 218 of the mobile device 200.

In various embodiments, the magnetic field image(s) and thehyperspectral image(s) may be taken in different order or concurrentlyfrom a same position in 3D space with respect to the target area. Insome embodiments, a determination is made whether the magnetic fieldimage(s) or the hyperspectral image(s) are sufficient to generate a 3Dimage rendering of the target area. If not, the diagnosis engine 242provides guidance on a user interface of the mobile device 200 as to howto position the mobile device 200 in 3D space relative to the targetarea of the patient.

In some embodiments, each of the images is further enhanced. In thisregard, for each image, the diagnosis engine 242 determines a quality ofthe resolution. If the quality of the resolution is below apredetermined threshold, then the image is enhanced by one or moretechniques such as deep learning models, as discussed herein.

At block 506, the diagnosis engine 242 generates a 3D image of thetarget area based on the received one or more magnetic field images andone or more hyperspectral images, coupled with the 3D positioninformation of each of the images.

At block 508, the diagnosis engine 242 determines the medical conditionof the target area based on the generated 3D image.

Example Computer Platform

As discussed above, functions relating to generating a 3D rendering of atarget area of a patient and a diagnosis of the medical condition basedthereon, as well as other functions discussed herein, can be performedwith the use of different types of mobile devices connected for datacommunication via wireless or wired communication, as shown in FIG. 1.An example mobile device 200 in the form of a smart phone was discussedin the context of FIG. 2. FIG. 6 is a functional block diagramillustration of a computer hardware platform such as a user device,patient database 110, reference database 112, or computing device of anauthorized medical professional 120 that can communicate with variousnetworked components.

The computer platform 600 may include a central processing unit (CPU)604, a hard disk drive (HDD) 606, random access memory (RAM) and/or readonly memory (ROM) 608, a keyboard 610, a mouse 612, a display 614, and acommunication interface 616, which are connected to a system bus 602.

In one embodiment, the HDD 606, has capabilities that include storing aprogram that can execute various processes, such as the diagnosis engine640, in a manner described herein. The diagnosis engine 640 may havevarious modules configured to perform different functions. For example,there may be a magnetic field image module 643, hyperspectral imagemodule 644, image reconstruction module 645, 3D positioning module 646,guidance module 647, and/or image analysis module 648. Each of thesemodules was discussed in detail before and will therefore not berepeated here for brevity. In one embodiment, one or more of thesemodules may be used to control a mobile device remotely over a network.Stated differently, one or more functions discussed in the context of auser device may be delegated to a remote computing device, therebyconserving the computational resources of the mobile device.

In one embodiment, a program, such as Apache™, can be stored foroperating the system as a Web server. In one embodiment, the HDD 606 canstore an executing application that includes one or more librarysoftware modules, such as those for the Java™ Runtime Environmentprogram for realizing a JVM (Java™ virtual machine).

CONCLUSION

The descriptions of the various embodiments of the present teachingshave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

While the foregoing has described what are considered to be the beststate and/or other examples, it is understood that various modificationsmay be made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that the teachings may beapplied in numerous applications, only some of which have been describedherein. It is intended by the following claims to claim any and allapplications, modifications and variations that fall within the truescope of the present teachings.

The components, steps, features, objects, benefits and advantages thathave been discussed herein are merely illustrative. None of them, northe discussions relating to them, are intended to limit the scope ofprotection. While various advantages have been discussed herein, it willbe understood that not all embodiments necessarily include alladvantages. Unless otherwise stated, all measurements, values, ratings,positions, magnitudes, sizes, and other specifications that are setforth in this specification, including in the claims that follow, areapproximate, not exact. They are intended to have a reasonable rangethat is consistent with the functions to which they relate and with whatis customary in the art to which they pertain.

Numerous other embodiments are also contemplated. These includeembodiments that have fewer, additional, and/or different components,steps, features, objects, benefits and advantages. These also includeembodiments in which the components and/or steps are arranged and/orordered differently.

Aspects of the present disclosure are described herein with reference tocall flow illustrations and/or block diagrams of a method, apparatus(systems), and computer program products according to embodiments of thepresent disclosure. It will be understood that each step of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the call flow illustrations and/or block diagrams, can beimplemented by computer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, special purpose computer, or other programmabledata processing apparatus to produce a machine, such that theinstructions, which execute via the processor of the computer or otherprogrammable data processing apparatus, create means for implementingthe functions/acts specified in the call flow process and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the call flow and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the call flow process and/or block diagramblock or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in thecall flow process or block diagrams may represent a module, segment, orportion of instructions, which comprises one or more executableinstructions for implementing the specified logical function(s). In somealternative implementations, the functions noted in the blocks may occurout of the order noted in the Figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or call flow illustration, and combinations of blocksin the block diagrams and/or call flow illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the foregoing has been described in conjunction with exemplaryembodiments, it is understood that the term “exemplary” is merely meantas an example, rather than the best or optimal. Except as statedimmediately above, nothing that has been stated or illustrated isintended or should be interpreted to cause a dedication of anycomponent, step, feature, object, benefit, advantage, or equivalent tothe public, regardless of whether it is or is not recited in the claims.

It will be understood that the terms and expressions used herein havethe ordinary meaning as is accorded to such terms and expressions withrespect to their corresponding respective areas of inquiry and studyexcept where specific meanings have otherwise been set forth herein.Relational terms such as first and second and the like may be usedsolely to distinguish one entity or action from another withoutnecessarily requiring or implying any actual such relationship or orderbetween such entities or actions. The terms “comprises,” “comprising,”or any other variation thereof, are intended to cover a non-exclusiveinclusion, such that a process, method, article, or apparatus thatcomprises a list of elements does not include only those elements butmay include other elements not expressly listed or inherent to suchprocess, method, article, or apparatus. An element proceeded by “a” or“an” does not, without further constraints, preclude the existence ofadditional identical elements in the process, method, article, orapparatus that comprises the element.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various embodiments for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments have more featuresthan are expressly recited in each claim. Rather, as the followingclaims reflect, inventive subject matter lies in less than all featuresof a single disclosed embodiment. Thus, the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separately claimed subject matter.

What is claimed is:
 1. A mobile device comprising: a processor; atransceiver coupled to the processor; an accelerometer coupled to theprocessor; a gyroscope coupled to the processor; a storage devicecoupled to the processor; a diagnosis engine software stored in thestorage device, wherein an execution of the diagnosis engine by theprocessor configures the mobile device to perform acts comprising:receiving one or more magnetic field images of a target area of apatient; receiving one or more hyperspectral images of the target areaof the patient; generating a three-dimensional (3D) image of the targetarea based on the received one or more magnetic field images, one ormore hyperspectral images, and positioning information from at least oneof the accelerometer or the gyroscope; and at least one of diagnosing ormonitoring a medical condition of the target area based on the generated3D image.
 2. The mobile device of claim 1, wherein receiving one or moremagnetic field images of the target area comprises: emitting a magneticfield by the transceiver onto the target area; and receiving at leastone of radio or magnetic signals from the target area in response to theemitted magnetic field of the transceiver.
 3. The mobile device of claim2, wherein the magnetic signals from the target area are received by amagnetic field sensor of the mobile device.
 4. The mobile device ofclaim 2, wherein receiving one or more magnetic field images of thetarget area further comprises, for each of the one or more magneticfield images: the diagnosis engine providing guidance on a userinterface of the mobile device as to how to position the mobile devicein 3D space with respect to the target area.
 5. The mobile device ofclaim 1, wherein receiving one or more hyperspectral images of thetarget area comprises, for each hyperspectral image: controlling a lightsource of the mobile device, to emit light at one or more predeterminedwavelengths; and recording a hyperspectral image of an anatomy of thetarget area by a camera of the mobile device.
 6. The mobile device ofclaim 5, wherein receiving one or more hyperspectral images of thetarget area further comprises, for each of the one or more hyperspectralimages: the diagnosis engine providing guidance on the user interface asto how to position the mobile device in 3D space with respect to thetarget area.
 7. The mobile device of claim 1, wherein at least one ofthe one or more magnetic field images and at least one of the one ormore hyperspectral images are taken concurrently from a same position in3D space with respect to the target area.
 8. The mobile device of claim1, wherein execution of the diagnosis engine further configures themobile device to perform acts comprising, for each hyperspectral image:determining a quality of a resolution of the hyperspectral image; andupon determining that the quality of the resolution of the hyperspectralimage is below a predetermined threshold, enhancing the image by way ofa deep learning model.
 9. The mobile device of claim 1, wherein the 3Dimage is further based on one or more photographs taken by a camera ofthe mobile device.
 10. A non-transitory computer readable storage mediumtangibly embodying a computer readable program code having computerreadable instructions that, when executed, causes a mobile device tocarry out a method, comprising: receiving one or more magnetic fieldimages of a target area of a patient; receiving one or morehyperspectral images of the target area of the patient; for each of theone or more magnetic field images and one or more hyperspectral images,tracking a three-dimensional (3D) position of the phone with respect tothe target are of the patient; generating a 3D image of the target areabased on the received one or more magnetic field images, the one or morehyperspectral images, and the corresponding tracked 3D position of thephone; and at least one of diagnosing or monitoring a medical conditionof the target area based on the generated 3D image.
 11. Thenon-transitory computer readable storage medium of claim 10, whereinreceiving one or more magnetic field images of the target areacomprises: emitting a magnetic field by a transceiver of the mobiledevice onto the target area; and receiving radio or magnetic signalsfrom the target area in response to the emitted magnetic field of thetransceiver.
 12. The non-transitory computer readable storage medium ofclaim 10, wherein the magnetic signals from the target area are receivedby a magnetic field sensor of the mobile device.
 13. The non-transitorycomputer readable storage medium of claim 10, wherein receiving one ormore magnetic field images of the target area further comprises, foreach of the one or more magnetic field images: providing guidance on auser interface of the mobile device as to how to position the mobiledevice in 3D space with respect to the target area.
 14. Thenon-transitory computer readable storage medium of claim 10, whereinreceiving one or more hyperspectral images of the target area comprises,for each hyperspectral image: controlling a light source of the mobiledevice, to emit light at one or more predetermined wavelengths; andrecording a hyperspectral image of an anatomy of the target area by acamera of the mobile device.
 15. The non-transitory computer readablestorage medium of claim 14, wherein receiving one or more hyperspectralimages of the target area further comprises, for each of the one or morehyperspectral images: providing guidance on the user interface as to howto position the mobile device in 3D space with respect to the targetarea.
 16. The non-transitory computer readable storage medium of claim10, wherein at least one of the one or more magnetic field images and atleast one of the one or more hyperspectral images are taken concurrentlyfrom a same position in 3D space with respect to the target area. 17.The non-transitory computer readable storage medium of claim 10, furthercomprising, for each hyperspectral image: determining a quality of aresolution of the hyperspectral image; and upon determining that thequality of the resolution of the hyperspectral image is below apredetermined threshold, enhancing the image by way of a deep learningmodel.
 18. The non-transitory computer readable storage medium of claim10, wherein the 3D image is further based on one or more photographstaken by a camera of the mobile device.
 19. A computer implementedmethod, comprising: directing a transceiver of a mobile device emit amagnetic field onto a target area of a patient; receiving a signal inresponse to the emitted magnetic field; creating one or more magneticfield images of the target area; controlling a light emitting source ofthe mobile device such that light is generated at one or more differentwavelengths; creating one or more hyperspectral images of the targetarea in response to the light generated at the one or more differentwavelengths; tracking a three-dimensional (3D) position of the mobiledevice with respect to the target area for each of the one or moremagnetic field images and one or more hyperspectral images; generating a3D image of the target area based on the created one or more magneticfield images, the one or more hyperspectral images, and thecorresponding tracked 3D position of the mobile device; and diagnosing amedical condition of the target area based on the generated 3D image.20. The method of claim 19, further comprising: for each of the one ormore magnetic field images and the one or more hyperspectral images,determining a quality of the image; and upon determining that the imageis below a predetermined threshold, enhancing the image by way of a deeplearning model.